The graphic images used in computer-generated displays have been stored in image memories at address locations mapping respective points at regular intervals along the raster scanning of a display image space. Each addressed location in image memory has contained a digital word, at least a portion of which has encoded the brightness, hue and saturation of a color picture element (or "pixel") at the corresponding point in image space (and, in run-length encoding schemes, the value of succeeding pixels). A number of different schemes for encoding the brightness, hue and saturation of color pixels exist in the prior art.
One may analyze each color pixel as the sum of the three additive primary colors, red, green and blue, for example. The amplitudes of the red, green and blue components may each be coded in a number n of bits, n normally being in the range five to eight inclusive. Coding may be linear, logarithmic, or in accordance with some other function. It is also known to linearly encode red, green and blue in different numbers p, q and r of bits depending on their relative contributions to luminance. Encoding green in seven bits, red in six bits and blue in five bits is an example of such coding, which provides luminance with nearly eight bits of amplitude resolution. There are 217 levels in luminance, assuming it to be comprised of 59% green, 30% red and 11% blue in accordance with the NTSC standard. This compares favorably with the 219 levels in luminance for eight-bit coding of luminance using level 35 as black and level 254 as white, as recommended by the Society of Motion Picture and Television Engineers (SMPTE). This amplitude resolution in luminance is generally considered to make quantization noise imperceptible to the human viewer. The reader is referred to M. F. Cowlishaw's paper "Fundamental Requirements of Picture Presentation" appearing on pages 101-107 of PROCEEDINGS OF THE SID, Vol. 26/2, 1985, for a comprehensive treatment of coding additive primary colors in differing numbers of bits.
One may analyze each color pixel as the sum of a luminance-only primary color and two chrominance-only primary colors. The luminance-only primary represents whiteness or brightness of the pixel. The chrominance-only primaries do not correspond with any real color, but together are representative of the difference of any real color from the luminance-only primary. Analyzing pixels in terms of luminance and two color-difference signals is usually done so that one may encode the two color-difference signals in less bandwidth than luminance. In digital sampled-data terms this usually corresponds to chrominance being sampled in image space at a subsampled rate compared to luminance being sampled in image space. Subsampling may be in the direction of line advance as well as in the direction in which scan lines extend.
A small computer may be used to decode television display matertial that has been encoded in an economical format (e.g., to permit the transmission of image data via telephone lines or the recording of image data on compact disc). This small computer may be provided with general-purpose memory, portions of which are available for use as image memory to provide buffering between an irregular flow of received image data and the regular flow of image data to the display. It is desirable to provide an image memory configuration that is well suited to being used interchangeably with other data storage in general-purpose memory and does not require the use of dedicated portions of the memory for image storage.
The encoding of television information for transmission over media of such limited bandwidths as those available from a telephone line or compact disc forces the designer to resort to powerful video compression methods These methods rely upon transmitting as little new image information per frame as possible and upon storing as much old image information as possible; and transmission of new image information cannot be done, at least not entirely, in real time. In order to write a display in real time, then, it is essential then to have frame buffer storage memory with the capability of storing at least two frames of video information. Such memory can be written to from a flow of compressed image data received in non-real time and read from so as to supply the display apparatus with a regular flow of image data in real time. The frames of stored information are updated on one frame and used to support display on the next, in staggered phasing. The frame buffer storage memory is bit-map-organized for convenience in constructing updated images from previous image data in accordance with instructions included in the compressed video data.
Image memories, the addressable storage locations of which map corresponding picture elements or "pixels" on a display screen and which store single bits descriptive of whether those corresponding pixels are bright or dark, have been described as being "bit-map-organized" for many years. In recent years the term "bit-map-organized" has been applied to certain image memories in which a pixel variable related to brightness is not expressed in terms of a single bit, but rather in terms of a plurality of bits. Such brightness-related variables may be luminance variables or may be color-difference variables used in connection with describing color displays, for example; or they may be component color variables, as another example. The term "bit-map-organized" has been extended to refer to two different memory configurations, each storing a plural-bit value descriptive of a pixel variable.
A plural-bit-variable bit-map-organized image memory of a first general type known in the prior art can be thought of as employing a number of planes, which number equals the total number of bits in the plural-bit-variable(s) describing a single pixel. The most significant bits of a first of the pixel variables are stored in the first bit plane at storage locations having respective addresses mapping respective pixel locations in the display; the next most significant bits of the first pixel variable are stored in the second bit plane at storage locations having respective addresses mapping respective pixel locations in the display in a manner corresponding to the mapping of the storage locations in the first bit plane; and so forth, proceeding to less significant bits in the first pixel variable, then proceeding through the bits of each other pixel variable (if any) proceeding from most significant to least significant bit. Responsive to a single address this type of memory furnishes simultaneously the respective plural bits of all the pixel variables descriptive of a particular pixel. Essentially, the spatial positions of individual pixels in the display have a one-to-one correspondence with respective image memory addresses, in a spatial mapping. This spatial mapping is held together by the tracing of the display screen and scanning of image memory addresses each being done in accordance with a prescribed pattern of correspondence between these activities. As long as the pattern of correspondence between these activities is adhered to, the rate at which and order in which these activities are carried out do not affect the spatial mapping between the image memory addresses and the spatial positions of display pixels.
A second general type of plural-bit-variable bit-map-organized image memory known to the prior art does not require a one-to-one correspondence between image memory address and the spatial positions of display pixels. There is a list of the values of the plural-bit pixel variables in a prescribed cyclic order, which cycles are arranged in the sequence of the tracing of the spatial positions of pixels in the display. The list is converted to a string of values of the pixel variables, with the bits in each value arranged in prescribed order according to relative significance. Each string of values is divided into words of given bit length, which words are stored respectively in successively addressed locations in the image memory. An image memory of this second general type has to be read out to a formatter with pixel unwrapping capability. The formatter reconstitutes the words into a string of values which are then parsed back into successive values of each pixel variable. The variables for each pixel are temporally aligned by the formatter to be available at the time the spatial position of that pixel is reached in the scanning of the display screen.
When a pixel is described in terms of plural variables--e.g., three primary color variables, or a luminance variable and two chrominance variables--it has been a general practice to group these variables in a prescribed order for each pixel and to use each group as subvariable components of a respective value of a complex pixel-descriptive variable. The values of this complex variable are then stored in a bit-map-organized image memory organized as either the first or the second type of image memory described above. This practice is reasonably satisfactory as long as the pixel-descriptive variables used as subvariable components of the complex variable are sampled at corresponding points in display space and with the same sampling density. However, it is desirable to be able to sample the pixel variables at differing sampling densities. This is primarily so that compressed video data can be more efficiently coded, but image memory is also conserved and image processing involves less calculation. If the pixel variables have different sampling densities in image space, using complex pixel-descriptive variables becomes unattractive.
A type of dual-ported, dynamic random-access memory that has recently become commercially available is the so called "video random-access memory" or "VRAM". This dynamic memory, in addition to a random-access input/output port through which information can be written into or read out of the memory, has a serial-access port from which a row of data can be read serially at video scan rates. The row busses of a principal dynamic random-access memory portion of a VRAM are arranged to transfer data in parallel to a smaller auxiliary memory of the VRAM, during an interval equal to the read interval from the random-access port. A counter is provided in each VRAM for scanning the addresses of the auxiliary memory during its reading, so the auxiliary memory can function as a shift register. After parallel loading of the auxiliary memory, its contents are read out serially through the VRAM serial output port, with the counter counting at a relatively high clock rate. This clock rate can be the rate at which the luminance-only picture elements are delivered to the display monitor of the computer apparatus, for example. This speed of reading is possible because the capacitance-to-substrate of the auxiliary memory busses is relatively low owing to the smaller size of this auxiliary memory.
In some types of VRAM, data can also be serially read into the auxiliary memory via the serial-access port, to be transferred in parallel into the principal dynamic random-access portion of the VRAM. This allows faster writing of the VRAM than is possible by writing information via its random-access port.
D. L. Sprague, N. J. Fedele and L. D. Ryan in their U.S. patent application Ser. No. 918,275 entitled "IMAGE STORAGE USING SEPARATELY SCANNED LUMINANCE AND CHROMINANCE VARIABLES", filed Oct. 14, 1986, and assigned to RCA Corporation, point out that time interleaving of lines of digitized chrominance information with lines of digitized luminance information is usefully applied to the reading of VRAM through its serial access port. Such practice facilitates the use of separate bit-map organizations of the second general type for luminance and chrominance variables in VRAM. The use of separate bit-map organizations for luminance and chrominance variables, Sprague et alii perceived, avoids the problems encountered in the use of complex pixel-descriptive variables in a unified bit-map organization when chrominance is sampled less densely in display space than luminance is. Sprague et alii describe in detail how to linearly pack image memory into VRAM, when separate bit-map organizations for luminance and chrominance variables are used. The use of separate bit-map organizations can be accommodated by using a rate-buffering memory for at least the chrominance samples, Sprague et alii realized. They used the rate-buffering memory for a further purpose to provide the sample bed information to support spatial interpolation in the direction transverse to line scan.
The coding of pixel information in terms of wideband luminance and two narrowband chrominance variables can reduce by nearly three times the bandwidth in terms of bits per unit time needed to linearly code image data, assuming these variables are sampled with the same numbers of bits, as Sprague et alii observed. This saves bandwidth in the compressed video data supplied to the computer and reduces the amount of computer memory needed for storing image data. However, there are problems with thus coding pixels in memory.
Coding pixel information in terms of luminance and chrominance is inefficient in terms of providing optimal resolution in luminance for given pixel code lengths. Amplitude resolution is needed more in luminance components of the drive signals to the display device than in color-difference components of the drive signals insofar as keeping the human observer unaware of quantizing noise is concerned. When the display color is white or shades thereof, the chrominance variables are non-contributory to the display. The resolution in luminance is defined just by the number of available bit places in the luminance variable alone, when pixel colors are defined in terms of luminance and chrominance variables.
Contrast this with the coding of color pixels in red (R), green (G) and blue (B) variables. If red and blue have the same number of bits defining their ranges of variation as green does, the resolution in luminance amplitude is determined by the number of bit places in green, the most significant component of luminance, divided by the fraction of luminance that is green. This fraction is customarily about 0.59. Eight bits in luminance can define 256 levels therein; 7 bits, 128 levels. Eight bits in green can define 434 levels approximately in luminance; seven bits in green can define 217 levels approximately in luminance; six bits in green can define 108 levels approximately in luminance. This advantage in luminance resolution still obtains when the number of bit places in red and in blue is reduced relative to the number of bit places in green, towards the number of bit places in the primary color being in proportion to the logarithm-base-two of their respective contributions to luminance, as advocated by Cowlishaw.
The I, Q or (R-Y), (B-Y) chrominance variables and Y luminance variable, used by Sprague et alii are not independent variables, as the additive color primaries red, green and blue are. Y and either pair of chrominance variables define a color space almost five times larger than that which is reproducible using the limited-brightness red, green and blue phosphors of a color kinescope. So, for a given pixel code length, one cannot define as finely that portion of color space in which physically realizable colors are located using the dependent variables, as one can using the additive color primaries. Further, the bit-compression of red, green and blue as independent variables towards greater equality of their respective contributions to luminance quantizing noise, as advocated by Cowlishaw, does not result in shortened bit-lengths for luminance and color-difference dependent variables. So the image data compression scheme employed by Sprague et alii in U.S. patent application Ser. No. 918,275 could not exploit the scheme of image data compression advocated by Cowlishaw.
(The concern with regard to efficiency in image coding arises primarily from the desire to implement video compression in the digital data supplied to the drawing processor of the computer, one should understand. Usually all calculations in the drawing processor take place at a rate determined by the time needed to process the pixel variable with most bits of amplitude resolution. If calculations are made concerning a pixel variable with fewer bits amplitude resolution, it is customary to pad the variable with added ZEROs. This is the easiest procedure when image calculations are carried out separately and serially on component portions of the image data. When image calculations are carried out separately and parallelly on component portions of the image data, padding variables to equal bit length is still the easiest procedure, since the calculating hardware may be simply replicated and then operated synchronously. Efficient image coding can allow denser packing of image memory, but the formatter following image memory must have a more sophisticated pixel unwrapping capability if dissimilar bit lengths are used in the various pixel variables. Especially as the cost of computer memory per bit falls with progress in the art of making random-access memory, it may be preferred to pad the pixel variables stored in image memory to even bit-lengths, where their bit lengths differ by only one or two bits, in order to simplify timing and formatting problems.)
It is more difficult to take into account the effects of inaccuracies in image calculations, as introduced by truncation and round-off, when one uses dependent variables in the calculations rather than the additive primaries. The dependent variables tend to be signed quantities, so the offsets introduced to make them always positive for purposes of calculation enter the multiplicative processes associated with the manipulation of pixels in space. The offsets can affect less significant bits of product terms, thereby to complicate truncation or round-off procedures. The tendency of successive errors in calcuation to be self-compensating, rather than cumulative in effect, is adversely affected by calculating in terms other than the additive primaries.
It is pointed out that knowledge gained from the designing of camera for broadcast color television provides a clue concerning how to advance the art with regard to the types of frame storage memory Sprague et alii indicate are to be favored for computer graphics--i.e., those that store component parts of color images in separately bit-map-organized portions of memory. In broadcast color television some cameras use three camera tubes which are respectively red-responsive, green-responsive and blue-responsive. The spatial registration of the three camera tubes is difficult to maintain with sufficient accuracy for all portions of the image such that a luminance signal formed from additively combining the responses of the three camera tubes in theoretically correct proportions will be accurate in the higher frequencies. The displacement among the three different color images in the camera tube responses causes their high frequency responses to fail to overlap in portions of the image. So, manufacturers of these color television cameras use the high-frequency response of only one of the camera tubes instead of the combined high-frequency responses of the camera tubes to define high-frequency luminance signal. Usually the high-frequency response of the green camera tube is chosen for this purpose, since green is the largest component of luminosity insofar as the human eye is concerned. Narrowband responses to the camera tube output signals so obtained by low-pass filtering are algebraically combined to obtain narrowband Y, I and Q signals--or, alternatively, narrowband Y, (R-Y) and (B-Y) signals--and the high frequency response from the one camera tube is added to the narrowband Y signal to define a wideband Y signal.
The present invention is based upon the reasoning that one can combine in other ways the four basic pieces of image information, namely, the narrowband red signal, the narrowband green signal, the narrowband blue signal, and the detail signal in a spatial-frequency band above the lower-spatial-frequency baseband shared by the three narrowband color signals. The detail signal can be admixed with one of the narrowband color signals to create a wideband signal which together with the other two narrowband color signals will contain the image information a human observer uses. ln the display apparatus the detail signal can be extracted from the wideband signal by high-pass spatial filtering. The extracted detail signal can be used to adjust the high-spatia-frequency content of the wideband signal, if this is necessary. The extracted detail signal is also admixed with the two narrowband color signals in proportion with the contributions of their respective colors to luminance, thereby to form additional wideband color signals. The three wideband color signals are then used to display the original image in a mixed-highs presentation.
The wideband signal containing detail information may simply be a wideband green signal, for example. This can support a mixed-highs presentation on a display monitor with color kinescope that is similar to that generated from Y, I, Q supplied from a three-camera-tube television camera tube. This is known to be commercially acceptable.
As another example, the wideband signalcontaining detail information may be a narrowband green signal added together with luminance detail formed in response to red, green and blue portions of image. lf this is done, it is convenient to reduce the luminance detail in proportion to the contribution made thereto by the green component of luminance. This avoids the need for having to adjust detail level in the wideband signal in order to make it an appropriate green drive signal for the kinescope. Luminance detail is afforded somewhat less amplitude resolution than lower-spatial-frequency luminance when this is done. Seven bit resolution for narrowband green allows 217 levels for low-spatial-frequency luminance, as noted previously. However, only 0.59 times the 128 levels afforded by seven bit resolution are available for high-spatial-frequency luminance coded together with narrowband green. This is about 75 levels of luminance detail resolution.
The literature indicates that 32 levels of luminance detail resolution in linearly coded video is all that is needed to support displays that the human viewer cannot distinguish from broadcast-quality television displays. The reader is directed to the paper by D. E. Troxel et alii entitled "A TWO-CHANNEL PICTURE CODING SYSTEM: I - REAL TIME IMPLEMENTATION", which appeared on pages 1841-1848 of IEEE TRANSACTIONS ON COMMUNICATIONS, Vol. COM-24, No. 12, December 1981. They indicate five bits per sample can linearly code luminance highs so pictures reconstructed from digitized samples are indistinguishable from broadcast-quality originals irrespective of viewing distance. If viewing distance is at least four times picture height, four bits can suffice.